{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Credits: this code and documentation was adapted from Paul Butler's [sklearn-pandas](https://github.com/paulgb/sklearn-pandas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to use a dataframe as a training set, but it needs to be converted to an array first. `DataFrameMapper` is used to specify how this conversion proceeds. For example, PCA might be applied to some numerical dataframe columns, and one-hot-encoding to a categorical column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformation Mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><tr><th></th><th>pet</th><th>children</th><th>salary</th></tr><tr><th>1</th><td>cat</td><td>4.0</td><td>90</td></tr><tr><th>2</th><td>dog</td><td>6.0</td><td>24</td></tr><tr><th>3</th><td>dog</td><td>3.0</td><td>44</td></tr><tr><th>4</th><td>fish</td><td>3.0</td><td>27</td></tr><tr><th>5</th><td>cat</td><td>2.0</td><td>32</td></tr><tr><th>6</th><td>dog</td><td>3.0</td><td>59</td></tr><tr><th>7</th><td>cat</td><td>5.0</td><td>36</td></tr><tr><th>8</th><td>fish</td><td>4.0</td><td>27</td></tr></table>"
      ],
      "text/plain": [
       "8x3 DataFrames.DataFrame\n",
       "│ Row │ pet    │ children │ salary │\n",
       "┝━━━━━┿━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━┥\n",
       "│ 1   │ \"cat\"  │ 4.0      │ 90     │\n",
       "│ 2   │ \"dog\"  │ 6.0      │ 24     │\n",
       "│ 3   │ \"dog\"  │ 3.0      │ 44     │\n",
       "│ 4   │ \"fish\" │ 3.0      │ 27     │\n",
       "│ 5   │ \"cat\"  │ 2.0      │ 32     │\n",
       "│ 6   │ \"dog\"  │ 3.0      │ 59     │\n",
       "│ 7   │ \"cat\"  │ 5.0      │ 36     │\n",
       "│ 8   │ \"fish\" │ 4.0      │ 27     │"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using ScikitLearn\n",
    "using DataFrames: DataFrame, NA, DataArray\n",
    "using DataArrays: @data\n",
    "@sk_import preprocessing: (LabelBinarizer, StandardScaler)\n",
    "\n",
    "data = DataFrame(pet=[\"cat\", \"dog\", \"dog\", \"fish\", \"cat\", \"dog\", \"cat\", \"fish\"],\n",
    "                 children=[4., 6, 3, 3, 2, 3, 5, 4],\n",
    "                 salary=[90, 24, 44, 27, 32, 59, 36, 27])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map the Columns to Transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mapper takes a list of pairs. The first is a column name from the DataFrame, or a list containing one or multiple columns (we will see an example with multiple columns later). The second is an object which will perform the transformation which will be applied to that column:\n",
    "\n",
    "Note: `ScikitLearn.DataFrameMapper` won't be available until `DataFrames` is imported"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mapper = DataFrameMapper([(:pet, LabelBinarizer()),\n",
    "                          ([:children], StandardScaler())]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference between specifying the column selector as :column (as a single symbol) and [:column] (as a list with one element) is the shape of the array that is passed to the transformer. In the first case, a one dimensional array with be passed, while in the second case it will be a 2-dimensional array with one column, i.e. a column vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `fit_transform!` shortcut to both fit the model and see what transformed data looks like. In this and the other examples, output is rounded to two digits with `round` to account for rounding errors on different hardware:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8x4 Array{Float64,2}:\n",
       " 1.0  0.0  0.0   0.21\n",
       " 0.0  1.0  0.0   1.88\n",
       " 0.0  1.0  0.0  -0.63\n",
       " 0.0  0.0  1.0  -0.63\n",
       " 1.0  0.0  0.0  -1.46\n",
       " 0.0  1.0  0.0  -0.63\n",
       " 1.0  0.0  0.0   1.04\n",
       " 0.0  0.0  1.0   0.21"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(fit_transform!(mapper, copy(data)), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first three columns are the output of the LabelBinarizer (corresponding to `cat`, `dog`, and `fish`\n",
    " respectively) and the fourth column is the standardized value for the number of children. In general, the columns are ordered according to the order given when the DataFrameMapper is constructed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the transformation is trained, we confirm that it works on new data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1x4 Array{Float64,2}:\n",
       " 1.0  0.0  0.0  1.04"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample = DataFrame(pet= [\"cat\"], children= [5.])\n",
    "round(transform(mapper, sample), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform Multiple Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformations may require multiple input columns. In these cases, the column names can be specified in a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@sk_import decomposition: PCA\n",
    "mapper2 = DataFrameMapper([([:children, :salary], PCA(1))]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now running `fit_transform!` will run PCA on the `children` and `salary` columns and return the first principal component:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8x1 Array{Float64,2}:\n",
       "  47.6\n",
       " -18.4\n",
       "   1.6\n",
       " -15.4\n",
       " -10.4\n",
       "  16.6\n",
       "  -6.4\n",
       " -15.4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(fit_transform!(mapper2, copy(data)), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple transformers for the same column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiple transformers can be applied to the same column specifying them in a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3x1 Array{Float64,2}:\n",
       " 1.0\n",
       " 2.0\n",
       " 3.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@sk_import preprocessing: Imputer\n",
    "mapper3 = DataFrameMapper([([:age], [Imputer()])]; NA2NaN=true)\n",
    "data_3 = DataFrame(age= @data([1, NA, 3]))\n",
    "fit_transform!(mapper3, data_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Columns that don't need any transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only columns that are listed in the `DataFrameMapper` are kept. To keep a column but don't apply any transformation to it, use `nothing` as transformer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8x4 Array{Float64,2}:\n",
       " 1.0  0.0  0.0  4.0\n",
       " 0.0  1.0  0.0  6.0\n",
       " 0.0  1.0  0.0  3.0\n",
       " 0.0  0.0  1.0  3.0\n",
       " 1.0  0.0  0.0  2.0\n",
       " 0.0  1.0  0.0  3.0\n",
       " 1.0  0.0  0.0  5.0\n",
       " 0.0  0.0  1.0  4.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapper3 = DataFrameMapper([\n",
    "     (:pet, LabelBinarizer()),\n",
    "     (:children, nothing)])\n",
    "round(fit_transform!(mapper3, copy(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we can combine features from a DataFrame, we may want to use cross-validation to see whether our model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Float64,1}:\n",
       "  -1.09\n",
       "  -5.3 \n",
       " -15.38"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@sk_import linear_model: LinearRegression\n",
    "\n",
    "pipe = Pipelines.Pipeline([\n",
    "     (:featurize, mapper),\n",
    "     (:lm, LinearRegression())])\n",
    "round(CrossValidation.cross_val_score(pipe, data, data[:salary]), 2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.4.5",
   "language": "julia",
   "name": "julia-0.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
